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Abstract 

In this paper we study a mean field model for discrete time, finite number of states, 
dynamic games. These models arise in situations that involve a very large number of agents 
moving from state to state according to certain optimality criteria. The mean field approach 
for optimal control and differentia l games (continuous state and time) was introduced by Lasry 



and Lions [LL06a, LL06b, LL07|. The discrete time, finite state space setting is motivated 
both by its independent interest as well as by numerical analysis questions which appear in 
the discretization of the problems introduced by Lasry and Lions. 

Our setting is the following: we assume that there is a very large number of identical 
agents which can be in a finite number of states. Because the number of agents is very 
large, we assume the mean field hypothesis, that is, that the only relevant information for the 
global evolution is the fraction 7r™ of players in each state i at time n. The agents look for 
minimizing a running cost, which depends on ir, plus a terminal cost V". In contrast with 
optimal control, where usually only the terminal cost is necessary to solve the problem, 
in mean-field games both the initial distribution of agents 7r° and the terminal cost V N are 
necessary to determine the solutions, that is, the distribution of players n n and value function 
V n , for < n < N. Because both initial and terminal data needs to be specified, we call this 
problem the initial-terminal value problem. Existence of solutions is non-trivial. Nevertheless, 
following the ideas of Lasry and Lions, we were able to establish existence and uniqueness, 
both for the stationary and for the initial-terminal value problems. We discuss in some detail 
a particular model, the entropy penalized problem. In the last part of the paper we prove 
the main result of the paper, namely the exponential convergence to a stationary solution of 
(7T°, V°), as N — > oo, for the initial-terminal value problem with (fixed) data ir~ N and V N . 



1 Introduction 

In this paper we study a mean field model for discrete time, finite number of states, dynamic games. 
These models arise in situations that involve a very large number of agents moving from state to 
state according to certain optimality criteria. The mean field approach for optimal control and 



differential games (continuous state and time) was introduced by Lasry and Lions [LL06a, LL06t 



LL07|. In the continuous state and time setting, mean field problems gives rise to Hamilton- 
Jacobi equations coupled with transport equations. The discrete time, finite state space setting 
is motivated both by its independent interest as well as by numerical analysis questions which 
appear in the discretization of the problems introduced by Lasry and Lions. The discretization of 
these models has been studied by I. Capuzzo-Dolcetta and Y. Achdou. 
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Our setting is the following: we assume that there is a very large number of identical agents 
which can be in a finite number of states. Each agent behaves individually and rationally, moving 
from state to state according to certain optimality criteria. Furthermore, its decisions are based 
solely on the following information, which is know by every agent, the current state, and the 
fraction of agents in each state. As in non-cooperative games, there may be interactions between 
the players in different states, as we will explain in more detail bellow. Because the number 
of agents is very large, we assume the mean field hypothesis, that is, that only the fraction 7r™ 
of players in each state i at time n is the relevant information for the global evolution. The 
mathematical justification of mean field models has been investigated extensively by Lions and 
Lasry in yet to be published papers and we do not address these issues in this work. 

Let d > 1 and N > 1 be natural numbers, representing, respectively, the number of possible 
states in which the agents can be at any given time, and the total duration of the process. Let 
7T° and V N be given d-dimensional vectors. We suppose that tt° is a probability vector, the initial 
probability distribution of agents among states, and that V N , the terminal cost, is an arbitrary 
vector. A solution to the mean field game is a sequence of pairs of e?-dimcnsional vectors 

{(n n ,V n ) ; < n < N} , 

where n n is the probability distribution of agents among states at time n and VJ 1 is the expected 
minimum total cost for an agent at state j, at time n. These pairs must satisfy certain optimality 
conditions that we describe in what follows: at every time step, the agents in state i choose a 
transition probability, Py, from state i to state j. Given the transition probabilities P,™ at time 
< n < N, the distribution of agents at time n + 1 is simply 

i 

Associated to this choice there is a transition cost Cij(n, P). In the special case in which only 
depends on 7r and on the ith line of P we use the simplified notation Cij(ir,Pi.). This last case 
arises when the choices of players in states j ^ i do not influence the transition cost to an agent 
in state i. Let ei(-7r, P, V) be the average cost that agents which are in state i incur when matrix 
P is chosen, given the current distribution 7r and the cost vector V at the subsequent instant. We 
assume that 

ei(7r, P,V)=J2 c ij(*, p ) p ij + Vj p ij- 
j 

Define the probability simplex S = {(qi, •••,%) ; qj > OVj, Ej=i 1j = !}• The set of d x d 
stochastic matrices is identified with § d . Given a stochastic matrix P 6 S> d and a probability 
vector q G S, we define V(P, q, i) to be the d x d stochastic matrix obtained from P by replacing 
the z-th row by q, and leaving all others unchanged. 

Definition 1. Fix a probability vector n G § and a cost vector V G M. d . A stochastic matrix 
P G § d is a Nash minimizer of e(ir, •, V) if for each i G {1, d} and any q G S 

et{ir,P,V)<ei(n,V(P,q,i),V). 

Definition 2. Suppose that for each it G § and V G R d there exists a Nash minimizer P G § d of 
e(ir, •, V). Let N > 1, ir° € S (the initial distribution of states), and V N G M. d (the terminal cost). 
A sequence of pairs of d-dimensional vectors 

{(ir n ,V n ) ; < n < N} 

is a solution of the mean field game if for every < n < N — 1 

vr = E 3 ^(7: n ,p n )p t n 3 + v 3 n+1 pr 3 

(i) 
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for some Nash minimizer P n E S d of e(ir n , •, V n+1 ). 

Until the end of this section we will assume that for all (tt, V) G § x R d there exists a unique 
Nash minimizer P o f e(7 r, P, U). Conditions which guarantee the uniqueness of a Nash minimizer 
will be studied in §3.2, Under the uniqueness of a Nash minimizer for e, we can define the 
(backwards) evolution operator for the value function 



g n (V)=e(ir,P,V), 
as well as the (forward) evolution operator for tt 

1Cv(tt) = irP. 



(2) 



(3) 



Since the operator Q v commutes with addition with constants, it can be regarded as a map from 
K d /IR to R d /R. Here R d /IR is the set of equivalence classes of vectors in R d whose components 
differ by the same constant. In R d /R we define the norm 



W#:= infill + A||, 



(4) 



In this paper we will regard 0^, depending on what is convenient, as both a map in K d as well as 
a map in R d /R. 

We have the compact equivalent form for ([!]) 



T n+1 = K V n+l(7f n ). 



(5) 



In this paper we will consider solutions to (|^) which satisfy initial-terminal value conditions, tt° 
(or tt ) and V N , as well as stationary solutions, that we discuss in what follows. 

Definition 3. A pair of vectors (fr, V) is a stationary solution to the mean field game if there 
exists a constant A, called critical value, such that 



V 



/Cy(7f). 



(6) 



We should remark that the first equation in (|6|) can be written in R d /R as Q W (V) = V. 
Therefore, solutions to (^|) can be regarded as fixed points of (£/„-, JCy) in R d /R x §. 

The structure of the paper is as follows: we first start, in §0 by listing our main assumptions, 
as well as explaining where they are needed in the paper. In §pfwe address the issue of existence, 
theorem [l], and uniqueness, theorem ^ of Nash-minimizing transition matrices. Some general 
properties of the operator Q are studied in In ifl we establish several results concerning the 
existence, theorem [| and uniqueness, propositions [Pf H an d theorem EL of stationary solutions to 
mean field games. The initial-terminal value problem is studied in §|6[ We show also existence, 
theorem |^, and uniqueness, theorem |(], of solutions for this problem. Both in the stationary and 
initial-terminal value p roblem the un i quenes s proofs use a version of the mononicity argument 
of Lasry and Lions in [ LL06a , LL06b| , LL07 |. In the last section we address the convergence to 
equilibrium and establish one of the main results of the paper, theorem (?], which states that, as 
we take the initial and terminal conditions far appart (n — ±N), the solutions at n = converge 
exponentially to a stationary solution. Throughout the paper we discuss with detail the entropy 
penalized problem For this model we give an independent proof of existence of stationary solutions, 
proposition^ and study the large entropy limit, proposition^. Another important example is the 
optimal stationary solutions, discussed in §5.7 which give a variational interpretation of a solution 
of certain mean field games in terms of non-linear programming problems, proposition 
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2 Main Assumptions 

In this section, for the convenience of the reader, we list the main assumptions that will be needed 
in the text. We list here only the assumptions that will be used in the main results or the ones 
which are repeatedly used in the text. A few other assumptions will be introduced later in the 
text and will only be used "locally" in the section they are stated. 

The first two assumptions will be used in theorem |], §[Tl], to establish existence of a Nash 
minimizcr of e. 

Assumption 1. For each ir 6 §, V G M. d , P G E> d , and each index 1 < i < d, the mapping 
q i > ej(7T, V(P, q, i), V), defined for q G S, and taking values on R, is convex. 

Assumption 2. The map P t— > ej(7r, P, V) is continuous for all i. 



Concern ing the uniqueness of Nash minimizers, addressed in §^2, we need the following defi- 



nition (sec [ HK ] for the motivation of this assumption) : 



Definition 4. A function g : R dxd -> R dxd is diagonally convex if for all P X ,P 2 G R d , P 1 ^ P 2 , 
wg have 

E(^-^)(^( pl )-^(^ 2 ))>o. 

With this definition we can state the next assumption: 
Assumption 3. Let 



de t (n,P,V) 

i\ r ) — 



dPij 



Then g is diagonally convex. 



Since diagonal convexity may not be the unique way to ensure uniqueness of Nash minimizers, 
it is convenient to add uniqueness as an assumption, which obviously holds under assumptions 
but which can also hold under other alternative hypothesis. 

Assumption 4. For each (tt, V) there exists a unique Nash minimizer P(jr,V) of e(jr, -,V). 

The uniqueness of P(tt, V) makes the operators ICy and Q„ well defined. Therefore, from §[5] 
on we will always suppose that assumption |i| holds, even without explicit mention. 
To establish continuity of P(w,V), §3.3, proposition [|, we need: 



Assumption 5. For each index 1 < i < d. ei : § x E> d x R d — > R is a continuous function. 

Denote by piy (P) the matrix we obtain from P by replacing its i'-th row by its i-th row, and 
leaving all other rows (including the i-th) unchanged. 

Assumption 6. There exists C > such that for all i and i' , and any ir G S, P G S d 



i 



\c ij {-K,P)-c i , j {-K, Pi<i ,{P))\Pi i < C. (7) 



Note that the previous assumption holds if Cy is bounded, for instance. 

Some of the results of the paper only hold for transition costs which have a special dependence 
on P. The next assumption will be required frequently: 

Assumption 7. The cost Cij(jr,Pi.) depends on 7r and, for each i, only on the i-th line of P. 

To establish uniqueness of solutions (§|5.4 theorem [?], §[T|, theorem ||), as well as to obtain ex- 
ponential convergence to stationary solutions (§^, theorem |7|) it is convenient to have the following 
assumption on the operator Q: 
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Assumption 8. There exists a constant 7 > such that 

* ■ - gav)) + 7T • (g n (v) - g*{v)) > 7 |k - ^ll 2 , 

for any V, V € K d and 7r, 7f G §. 

An example where this last hypothesis is satisfied is the following: 

c ij (ir,Pi.) = Wi{w)+c ij (P i .) 1 (8) 

where W is a monotone function, that is, 

{tt - 7f) • (W(tt) - W^(tt)) > 7||tt - ^|| 2 , (9) 

where 7 is a positive constant. For instance, the gradient of a convex function is a monotone 
function. In this case we have 



g„(Vm = Wfa) + min £(cy (Pi.) + y,)p y 



The special structure in (|3|) arises naturally in certain problems, see § 5.7 



To establish the uniqueness of the critical value we need the following assumption: 

Assumption 9. For any tt £ S, the operator V 1— » (^(V) satisfies the following property: for all 
V 1 , V 2 and any i 6 argmax(V' 1 — y 2 ) we Aave 

ure'i/i i/ie opposite inequality if i G argmin(y 1 — V 2 ). 



As it will be proved in §4.1, proposition [?], assumption |?] implies assumption ^. However, we 



leave it explicit to make easier the understanding of what follows. 

The following strict concavity of Q is important to establish uniqueness of stationary solutions 
and the exponential convergence to equilibrium. 

Assumption 10. For all tt G S and all V 1 , V 2 G R d we have 

n ■ {g^v 2 ) - g^v 1 )) + K^iv 1 -v 2 ) < -1AV 1 -v 2 \\%. 

This last assumption is a slightly stronger version of inequality (|l^) , which is a consequence 
of assumptions || and 0. 



A final hypothesis allow us to establish certain bounds (lemma |2| in §7.1) which are useful in 
proving the exponential convergence to equilibrium: 

Assumption 11. There exists K > such that for all tt, tt G S, and for any matrix P G S d 

\ Ct] {TT,P)-C t] {K,P)\<K. (10) 

Note that the previous assumption holds if Cy is bounded, for instance. 

3 The Transition Matrix 

In this section we discuss the problem of existence, uniqueness and continuity of the Nash equilib- 
rium transition matrix P. As we will see, this problem is non-trivial and requires, in the general 



1 § 
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3.1 Existence in the general case 

Theorem 1. Suppose that assumptions^ and^ hold. Then, for any pair of vectors ir and V there 
exists a Nash minimizer P of e(n, -, V). 

Proof. Given a stochastic matrix P, define Pi(P) to be the set of vectors in § given by 

Fi(P) = argmin^g e l {n 7 P(P, q, i), V). 

The set Pj(P) is non-empty and convex. Define P(P) = Pi(P) x F 2 (P) x ... x Pd(P), where 
we identify the cartesian product with the the set of all stochastic matrices where the i-th row 
belongs to Fi(P). Clearly, F(P) is convex for all P. Furthermore, as we argue next, the graph 
{(P; F(P));P e S d } is closed. Indeed, suppose that P n -> P° and take Q n € P(P"), if Q n -» Q° 
we want to show that Q° S F(P°). Fix i and call q™ the i-th coordinate of Q n , then by hypothesis 
ei(w,V(P n ,qf,i),V) < ei(ir,V(P n ,q',i),V) for all q 1 £ S. As qf -» q° and P" -> P° we have 
that P(P", 9j n ,i) -> V(P°,q9,i) then we get that e 4 (vr, P(P°, g°, i), 7) < e 2 (7r, P(P°, g', z), V) for 
all g' G S. 

Then, because for each P, F(P) is a convex set and the graph (P, P(P)) is closed, we can 
apply Kakutani's fixed point theorem, which implies the existence of a matrix P that belongs to 
F(P). Thus P is a Nash minimizer of e(7r, ■, V). □ 



3.2 Uniqueness for diagonally convex costs 

As shown in the previous section, if assumptions |l| and || hold, for each tt and V there exists a 
transition matrix P which is a Nash minimizer of e(7r, •, V). In general, such minimizer may fail 
to be unique. Under the diagonally convex assumption ^ we will show uniqueness. 



Theorem 2. Suppose assumptions hold. Then there exists a unique transition matrix P which 
is a Nash minimizer of e(7r, •, V). 

Proof. Note that if P is any Nash minimizer of e(n, ■, V), its i-th line solves the constrained 
optimization problem 

min ei{TT,V{P,q,i),V) , 
ft >0, 

5> = i. 



Thus, if P 1 and P 2 are two Nash minimizers, they satisfy the KKT conditions | Pcd04| 

/' - im = 0, k = 1,2. 



^(7r,P fc ,U) k k 
JJJT, '' " 



]Tp,-i = o, 

3 

and 6ij Pij = , 

where i/j is the Lagrange multiplier associated with y\ Py = 1 and #y > corresponds to the 
constraint P^ > 0. From the first of the three equations above we conclude that 

Therefore, using the diagonally convex property, we have 



G 



which implies, when we use 0£-P| = 0, that 

Ef-PV - P 2 v 2 + P x v 2 + P 2 v x + P 1 ^ 2 - + P 2 ^ 1 ) < 

ij 

Now we can use that 

Y,P^\ = v l Vl</,fc<2, 
3 

to get 



E(4^+4^)<°- 



Since Piffle Pfj^h > O7 we obtain a contradiction. □ 

3.3 Uniqueness and continuity 

Suppose assumption |J holds. Consider the map which associates to each pair (n, V) its unique 
optimizing transition matrix P(n, v). Is it natural to ask whether this map is a continuous function. 
This is addressed in the next proposition. 

Proposition 1. Suppose assumptions hold. Then P(w,V) is a continuous function of tt and 
V. 

Proof. Consider sequences ir n — > n and V n — » V. The corresponding sequence of Nash minimizers 
P n = P(7r„, V n ) converges to a Nash minimizer, by the continuity of e, assumption g. Therefore, 
by the uniqueness hypothesis (assumption [|) P(ir n , V n ) — ► P(tt, V). □ 

3.4 The entropy penalized model 

Now we consider a special example, the entropy penalized model. We fix a positive constant e, 
and consider assume that Cij{-K,Pi.) — Cij{~K) + eln(Py), where cy is a continuous function of tt. 
For simplicity we will drop the ~. We have 



,(tt, P, V) = E Py (cy (tt) + e In Py + V } ) . (11) 



The term ePylnPy, with e > 0, is related to entropy and forces the agents to diversify their 
transition choices by enforcing a penalty if they do not do so. 

It is easy to prove that there exists a unique Nash minimizing transition matrix given by 



e 



P^V) = - (12) 

Efe e 

Also, this transition matrix is a continuous function of n and V. 

Now we present a useful formula for the second derivatives of Gw(V)i- 

d 2 g^(V)i _ PkPi -PkSki _ 7 
dV k dVi ~ e - kh 

where 

Pk = „ 6 c^w + v ^- - ( 13 ) 

E ro e 

Because ^ fc = 0, the matrix J^i has a zero eigenvalue, which is a reflection of the fact that 
Gti commutes with addition of constants. As we show in the next proposition this eigenvalue is 
simple. 
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Proposition 2. Suppose < pk < 1, J2kP k ~ a7 ^ ^ 

PfcP; -PkSki 



Jkl 



Then /las simple multiplicity. 
Proof. Observe that 

where 
and 



J=-D{Q-I), 
e 



D = diag{pi, ...,p n } 



Qu = 



If is not a simple eigenvalue, it would mean that there exists w and v which are linearly in- 
dependent eigenvectors corresponding to this eingenvalue. But then v and w are eigenvectors of 
Q corresponding to the eigenvalue 1. But this contradicts Perron- Frobenius theorem because the 
eigenvalue 1 is a simple eigenvalue of Q. □ 

4 Properties of Q 



In this section we discuss the main properties of the operator Q. In §4.1 we show that assumption 
|^ is a consequence of assumptions |] and 0, and in §4.2 we study concavity properties of Q. A- 



priori bounds, essencial to establishing existence of stationary solutions are considered in §4.3 and 



finally, in § 4.4 we prove strict concavity of Q for the entropy penalized model with two states. 



4.1 Assumption ^ 

Proposition 3. Suppose assumptions and [t| hold. Then assumption |^ holds. 

Proof. Let V £ M. d , k = 1,2. Let i £ argmaxF 1 — V 2 . By adding a constant, we may assume 
that V± = V 2 , and so because Q commutes with the addition of constants, it suffices to check that 

Because i is a maximizer of V} — V 2 , for all j we have V} — V 2 < 0, that is V} < V 2 . Let P? 4 be 

3 3 1 J 3 3 — ' 3 — 3 % 3 

such that 

3 

Then 

q,{v% < E c v ■■(*, p t) p l + v 3 lp l ^ E p ^ p l + v 3 2p l = sav% 

3 3 

Arguing similarly we obtain the opposite inequality when i £ argmin V 1 — V 2 . □ 

4.2 Concavity 

If assumption |?] holds, for each fixed index i the mapping 

v i * g^v)i 



is concave since 



g v (V)i = min E Cij(n, P^Py + V^, (14) 



3 
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is a pointwise minimum of linear functions of V. Furthermore, since tt > 0, 
is also concave. 

Suppose that P(ir, V), the transition matrix that realizes the minimum in (fl4|), is differentiable 
with respect to V . We will use the notation P^> v = P(n, V). Then 

dQjy)j = -n.v 
dVj 13 ' 

In general, however, P*> v is may not be differentiable. Nevertheless, we have the following rigorous 
statement: 

Proposition 4. Suppose assumptions and hold. Then 

(a) IfV 1 and V 2 are any two given vectors, we have 

GAv% < qav^ + Y^p^'^v 2 V 1 ^. (15) 

3 

(b) IfV 2 > V 1 , then 

3 

Proof, (a) Observe that 

j i 
(b) We just need to prove the first inequality. For this, note that 

gav% = E p Z''*(y? + ^fa ^ 2,7r )) 

□ 

Note that, by multiplying ([l5|) by tti and adding, we obtain 

7T • (^(^ 2 ) - GAV 1 )) - (V 2 - V 1 ) ■ /C y i (tt) < 0. (16) 

Since Q commutes with the addition of constants it is not possible to establish a strict concavity 
estimate like 

tt . (g^v 2 ) - g^v 1 )) + (v 1 - v 2 ) ■ k, v ^) < -^wv 1 - v 2 \\ 2 

However, in certain cases it is possible to obtain the following strict concavity estimate: 

it ■ (g^v 2 ) - g^v 1 )) + (v 1 - v 2 ) ■ k^) < -j^v 1 - v 2 \\%. 



In §4.4 we will give an explicit example. 
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4.3 A-priori bounds 

In the next proposition we give some a-priori bounds which are essencial to establishing the 
existence of fixed points (theorem |J) . 

Proposition 5. Suppose assumption |^ holds. Then for any (n, V) G § x R d and all indices i,i' 
we have 

\Q«(v)i-g v {v) t ,\<c. (17) 

Proof. Fix 7r and V let (Gir,ICv) be as in (||) and (||). Let P be the optimal transition matrix such 
that 

s*(y)i = °yfa p ) p ^ + ^ p «- 

3 

Since replacing the i'-th line by the i-th line in P yields a sub-optimal choice, we have 

G«{V)v <Y,a-i^,Pi,i'{P))Pio+VjPiv 

3 

Hence 

G*(V)i> -GAV)i < l][-^fa p ) + c ^fa^'( p ))] p «- 

3 

If we exchange the role of i and i' we have the desired estimate. □ 

4.4 Strict concavity for the entropy penalized model 

To show that the entropy penalized model satisfies the strict concavity property of assumption 
[Tc| , it suffices to show that the restriction of the linear form given by the matrix Dy 2 GTr(V)i to 
the space of vectors X 6 M d with ^ fe Xk = is uniformly definite positive for each i. This 
holds because of proposition ||, which states that the eigenvalue is simple, and the corresponding- 
eigenvector is Y — (1,...,1). The uniformity follows from the a-priori bounds in the previous 
section, which allow us to assume that ||V||# is bounded. In fact (uniform) strict concavity only 
holds for bounded ||V]J#- However, for the purposes of this paper this is enough because of the 
a-priori bounds in §4^. Thus if cy is bounded and ||V||# is bounded we have < pk < 1, and so 
assumption [l^ holds. 

5 Stationary Solutions 

In this section we study stationary solutions to mean field games. After the characterization of the 



critical value, in §5.1, as the average cost for the population of agents, we address the question of 
uniqueness of the critical value A for which (^|) admits a solution. In §5.2 we give an example where 



A is non- unique. However, after addressing the issue of existence of stationary solutions (in §|5.3|) , 
we revisit the uniqueness problem in §|5.4| giving conditions which imply uniqueness A, tt and V. 



These conditions are variations of the monotonicity conditions in [LL06a|. The entropy penalized 



model is revisited in §5.5, and the large entropy limit is considered in §5.£, where we establish 
uniqueness of stationary solution (proposition |l^) . This uniqueness proof uses a strong contraction 
argument and is thus suitable for the numerical approximation of large entropy penalized mean 
field games. We end this section, in §|5.7j, with a discussion of optimal stationary solutions, where 



certain variational problems give rise to mean field games (see [LL07], §2.6, for related problems) 



5.1 Representation of the critical value 

We will now give a representation formula for the critical value as the average transition cost. 
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Proposition 6. Suppose assumption [| holds. Let (jr, V) be a stationary solution to (|6|) 7 and A 

the corresponding critical value. Let P be the optimal transition matrix. Then 

A = ^2,%iC ij {Tt,P)Pi j . 

ij 

Proof. For each 1 < i < d, 

a = E - (vt - E ^ j • ( 18 ) 

Note that Cy(7f, P)P%j can be seen as the expected cost of transition agents that are in state 
i will have when moving to other states. If we multiply (fL8|) by 7fj and add, for 1 < i < d, we get 



«,J 1,3 



Let /i ,rP denote the probability measure on the set {l,2,...,d} 2 given by /z^ p = TTiPij. Since 
7f = 7fP, we have 

z i 

Therefore 

E - vj) tiPij = E - ^) = °- 

So 

a = E^(^^)mS p - 



□ 



5.2 Non-uniqueness of the critical value 

In this section we show that the critical value may not be unique. Consider the following example: 
Cij(w) given by 

Cl2 = c 2 i = 100, 

and 

cu(n e )=c 22 (7r e ) = 9. 
where n 9 = (6,1- 9), for < 6 < 1. Then F e = (0, 0), A 9 = 0, tt" and 



P° = 

is a stationary solution. 



1 
1 



5.3 Existence of stationary solutions 



Theorem 3. Suppose that assumptions U-|oj hold. Assume further that there exists C such that for 
any (tt, V) £ S x M. d and all indices i, i' we have (|l7|). Then there exists a pair of vectors (n, V), 
a constant A and a transition matrix P such that for all i, 

e*(y)i = E c v p ) p ij + VjPn = Vi + a, 

3 

and tt = ttP. 
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Note that ( |l7| ) will hold, by proposition if we assume additionally assumption || 

Proof. Since |}|| hold, by proposition [l], the optimal transition matrix P(tt, V) is a continuous 
function. Therefore the operator Gk(-) ■ — > R d /R is continuous. Furthermore, by estimate 
(0), ||g^(V)|| # is uniformly bounded for any V E R d . 

Consider the mapping (G n (V),]Cv(^)) : (R d /M) x § -> (R d /R) x S. By Brouwer's fixed point 
theorem this mapping has a fixed point, which is a stationary solution since 

[g 9 {v) = x+v 

|/Cy(7f) = 7f. 

□ 

5.4 Uniqueness of stationary solutions 

Now we address the problem of uniqueness of stationary solutions. The results in this section use 



the monotonicity methods introduced in [LL06a, LL06b, LL07| - in this discrete setting, different 



versions of the hypothesis will yield several uniqueness results. Under assumption |8| we will prove, 
in proposition |?], the uniqueness of stationary distribution tt. The uniqueness of critical value is 
established in proposition ^, using assumption ^, and finally under assumption [l^ we obtain the 
uniqueness of tt, V and A in theorem ||. 

Proposition 7. Suppose assumptions ^, and hold. Let (n k ,V k ) , k = 1,2, be stationary 
solutions: 

K vk {iT k ) = iT k , g 7lk (v k ) = \ k + v k 

where \ k are constants. Then tt 1 = tt 2 . 
Proof. From the hypothesis we have 

=(V X - V 2 ) ■ (JC v i (tt 1 ) -tt 1 - K,v 2 (tt 2 ) + tt 2 )+ 

+ (tt 1 - TT 2 )((g^(v 2 ) v 2 ) (g^v 1 ) - v 1 )) + (x 1 x 2 ) YJfo 1 - tt 2 ). 

i 

Note that the last term vanishes since Ylifai ~ ^i) = 0- Rewriting we have 
=7T 1 • (&X (V 2 ) - G nl (V 1 )) + (V 1 - V 2 ) ■ Kyi (tt 1 )+ 

+ tt 2 ■ (g^v 1 ) - g^(v 2 )) + (v 2 - v 1 ) ■ k v ,(tt 2 )+ 
+ ^ ■ (g^ (v 2 ) - g„i (v 2 )) + tt 2 - (v 1 ) - g n * (v 1 )). 

By proposition || and ( |l6|) , the first term above satisfies 

tt 1 • (g„i (v 2 ) - g^v 1 )) + (v 1 - v 2 ) ■ iCy^ 1 ) < o, 

and similarly for the second term. 

To analize the third term, observe that, by assumption^, we have 

n 1 ■ (g w 2 (v 2 ) - g^i (v 2 )) + tt 2 • (g^v 1 ) - Q^v 1 )) < -71k 1 - * 2 \\ 2 - 

Therefore, the estimates above imply 7H7T 1 — tt 2 \\ 2 < 0. □ 
Now we establish the uniqueness of the critical value: 

Proposition 8. Suppose assumptions 0, [?| and ^ hold. Let tt G S, X k e R and V k € R d , k = 1,2 

be solutions of 

g w (v k ) = X k + V k . 

Then X 1 = X 2 . 
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Proof. Choose i £ argmaxV 1 — V 2 . Then 

a 1 = g^v 1 ^ - v t x < gav% - V 2 = A 2 . 

By choosing i G argminT^ 1 — V 2 we obtain the opposite inequality, which then implies A 1 — A 2 . □ 

Therefore, under assumptions || and ^, we have both uniqueness of the stationary distribution 
7r and critical value A. We now address the uniqueness of the stationary value function V. 

Theorem 4. Suppose assumptions^, 0, || and hold. Let (ir k ,V k ), k = 1,2, be stationary 
solutions: 

K v u{-K k ) = -K k , g„k{v k ) = x k + v k 

where X k are constants. Then 

(a) IT 1 = 7T 2 .. 

(b) V 2 = V 1 + k, where k is a constant vector, 

(c) Ai = A a . 

Proof. If we follow the proof of Proposition |?], we can use assumption |l^ to get 

= -Till^ 1 ~ V 2 \\l - l2 \\V l - V 2 \\l - tIItt 1 - n 2 \\ 2 . 
This implies items (a) and (b). To get item (c), we observe that 

v 2 + a 2 = g^v 2 ) = g^v 1 + k) = g^v 1 ) + k = v 1 + a 1 + k = v 2 + a 1 , 

where ir = tt\ = 7T2, in the first and fourth equalities we used C?7r(^ fe ) = V k + \ k , in the second and 
fifth we used item (b), and in the third we used the fact that g^ commutes with constants. □ 

5.5 Entropy penalized stationary solutions 

Now we consider the entropy penalized model We will present a simple proof of existence of 
solutions that relies on the special structure of the problem. A simple computation yields 



9«(y)i = -eln 



E 



e 

k 



We will suppose further: 

Assumption 12. The function aj (tt) is continuous. 

Proposition 9. Suppose assumption\l^ holds. Consider the entropy penalized model (|ll|). Then 
there exists a pair of vectors (tt, V) 6 § x R d , a constant A € 1 such that 

g*{v) = \ + v 

and 7f = ttP = JCyijr). 

Proof. Define the strictly positive linear operator that associates to each vector tp S K d the vector 

e 

k 



Let e ' be the largest eigenvalue of the operator C w (ip) and tp™ the unique normalized eigenvector 
associated to e « , i.e., 
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By Pcrron-Frobenius Theorem, ip n is a strictly positive vector which is a continuous function of 

7r. We can define V* as ip'fc — e~~ . Let £[<j>]j = e~~ be the exponential transformation. These 
operators are related by 

£tt o £ = £ o . 



Hence 



(19) 



Define a new probability vector 



Thus we have defined a operator JC : S — > § which is continuous. By Brower's fixed point theorem, 
/C has a fixed point 7f. Define y x and as above, and Py = — - 



then (|19|) holds. □ 



5.6 Stationary Solutions with large entropy 



The construction of fixed points for the entropy penalized model in the last section depends on 
Brower's fixed point theorem. In the case of large entropy we can use a contraction argument to 
establish the existence of a stationary solution. Before proving and stating this result we need an 
elementary lemma 



Lemma 1. Let T 



jm x jgin ^ e a qX ma ppi n g^ Suppose 
DT = 



Ex M 
Ei E 3 



where E^, M are n x n matrices. If \\M\\ is bounded and ||-Efc|| is sufficiently small then T 2 is a 
strong contraction in E> d x R d /R. 



Proof. It suffices to observe that 

can be written as 

D(T 2 ) = 

and therefore ||£>(T 2 )|| < 1. 



D(T 2 )(x) = DT(T{x)x)DT{x) 





M ' 




' E x 


M ' 




Ei Ei - 


±ME 2 


ExM - 


\- ME 3 ' 


. E 2 


E 3 _ 




E 2 


E 3 




E2E1 - 


- E3E2 


E 2 M-{ 


- E 3 E S 



□ 



To establish the main result in this section we need to replace assumption [12] by 
Assumption 13. The function Cij(ir) is a C 1 function. 

Proposition 10. Suppose assumption ^ holds. Then, for large e, there is a unique stationary 
solution. Additionally, let T(ir,V) — (/Cy- )e (7r), Q V (V)), then T 2 is a strong contraction. 

Proof. From proposition || we have ||^ T (V)||,jt is uniformly bounded. Therefore it suffices to show 
that the operator (IC, Q) is a strong contraction for V in a compact set (with respect to the norm 
|| • ||#). Also, we can replace Q by 
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and K-v(tt) by 



Since any fixed point to is a stationary solution. In this way ||£ ir (V)||# = and for 

any tt (not necessarily a probability measure), fCvM is a probability measure, and agrees with 
K,y (tt) if tt is a probability measure. 

We will show that, for e sufficiently large, the pair (Q, JC) satisfies the hypothesis of lemma |l|. 
To do so, we first compute the matrix 

ag ag 
ale d\ 

dV On 

We will show that, when e — > oo, this matrix converges to 

[0] [l/d] _ 

where we denote by [a] the d x d matrix whose entries arc all identical to a. 



Since Q n ,e(V)i = -eln J2 



, we have: 



dGjy) 

dV 



when e — > oo. We also have 



/or fv\\ T Sc ih M "n^+ v i 
( oQjy)\ _ 2^k dnj e 



when e — > oo. 

Now we consider K,v(ir)i = J2k n kPki, where 



We have 



l/d, 



1 ^ ggifc(7r) 



9XV(tt) 
diTi 



k 



El 



h iW+Vi c _c fci W + Vj f dc kl (it) _ dCki(lT) 



If we take e — ► oo, the second term tends to zero while Pij -> i. Thus 



97T,- 



1 

d 



Since 



we obtain 



fc 



dV* 
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when e — > oo. 

Finally, observe that 



ae 


ag 




ay 






dK 


dK 




dV 







I-[l/d] [0] 
[0] I-[l/d] 



This shows that, in the limit e — > oo, we have 



ag ag 
m. at 

dV 9tt 



where 



ae ae 
av a-* 




' [0] 


M ' 


dK. dK 

av a* 




. PI 





M = {I-[l/d])[W d -^] 



Thus, for e large enough lemma |T] yields the strong contraction property of T 2 



□ 



5.7 Optimal Stationary Solutions 

Given a probability measure r]ij, 1 < i,j < d, define 7r^ = J^ffe and let P^ be a stochastic 
matrix such that r/ij — 71^ Py. If tt 17 never vanishes then P r ' is uniquely defined by Py = ^f. A 
probability measure ryy is stationary if 

X>i=I>- (20) 

i 3 

For our purposes, in this section it is convenient to consider the following two auxiliary assump- 
tions: 

Assumption 14. For each 1 < i < d the mapping Pj. > y\ cy(P,.)Py zs convex. 
and 

Assumption 15. TTie mapping r\ \— > j 7r '' c *i(-^ > i'?)^y * s strictly convex. 
Consider a C 1 convex function function / : M d — > M. Consider the problem 

min^<' Cu (P")P^+/(^), (21) 

ij 

where the minimum is taken over all probability measures r\ satisfying (p0|). 

Proposition 11. Let r\ > be a solution of (fil|). Let V v S IR" 1 6e i/ie Lagrange multiplier 
associated to the constraint (|2C|), and A 1 ' i/ie Lagrange multiplier corresponding to J^y Jfaj = 1. 
T/zen (7T 1 ', V 7 ) is a stationary solution of 

V" + A ? ' = ^(V"), 

^(V)i = |^(tt) + min V cy (p.)Py + PijV}. 

i 

Proof. Let 77 be as in the statement. As before write ?7y = rfPPj. Then both ir n and PPj are 
critical points of the functional 

/(7T)+$>i 



v n p- 



VP - A" 



(22) 
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Consequently, for each i, Pj. is a critical point of 

^2c ij (P)P ij +Vj>P ij , 



which by the convexity hypothesis, assumption 14, is a minimizer. Furthermore, by differentiating 
(|22j ) with respect to 7Tj, we obtain 



Finally, we can write (20) as 

which ends the proof. □ 

Proposition 12. Suppose assumptions^, and\jl^ hold. Then there exists at most one stationary 
solution with 7T, P > 0. 



Proof. This proposition follows from the well known fact (see for instance [ Ped04 ) that for strictly 
convex objective functions under linear constraints the KKT conditions are not only necessary 
but also sufficient. □ 

Note that if / is a strictly convex function, then the previous proposition gives us another proof 
of the existence and uniqueness of the stationary solution in the case cy (tt, P) — 5y (Pi. ) + W(w), 
wi th W (n) = J£(tt). 

6 Solutions to the Mean Field Game initial-terminal value 
problem 



In this section we prove the existence (§6.1) and uniqueness (§p.2j) of solutions to the initial- 
terminal value problem. 

6.1 Existence of Solutions 

Theorem 5. Suppose assumptions ^ and hold. Then for any initial probability vector jt G § 
and terminal cost V there exists a solution 

{(n n ,V n ) ; < n < N} 

to the initial-terminal value problem for the mean field game with 7T° = 7T and V N = V. 

Proof. Suppose we are given a sequence 7r"'° £ § JV+1 of probability vectors, with 7r 0,0 — jr. Define, 
for < n < N 

V n '° = G^,o(V n+1 '°), 

with V N '° = V. Then let 

with TT ' 1 = TT. This procedure defines a continuous mapping from § w+1 into itself that associates 
to the sequence 7r n '° the new sequence of probability vectors it™' 1 . Therefore, by Brower's fixed 
point theorem, it has a fixed point, which corresponds to a solution to the problem. □ 



17 



6.2 Uniqueness 

As for stationary solution we adapt Lasry and Lions monotonicity arguments to obtain uniqueness 
of solutions. 

Theorem 6. Suppose assumptions^, Q <md| hold. Let {(tt™, V n ) ; < n < N} and {(% n , V n ) ; < 
n < N} be solutions of the the mean field game with tt° = n° and V N — V ■ Then ir n — TT n , and 
yn = yn ; f Qr all < n < N . 

Proof. We have 

g„ n {V n+1 ) = V n , JC vn+1 (-jr n ) = 7r n+1 , 

and 
Then 



g^(V n+1 ) = V n , JCy n+1 (7r n ) = n n+x . 



n=Q 
N-l 

+ J2(7r n - n n ) ■ [(g f n(v n+1 ) - v n ) - (g^(v n+1 ) - v n )]. 



71=0 

Note that (V N - V N ) ■ (n N - n N ) = and (tt - tt ) • (V° - V" ) = 0. Thus rewriting the identity 
above we have 

N-l 

= * n ■ (G^(V n+1 ) - 5x»(r +1 ))+t v ,. + i(i") • {V n+1 - V n+1 ) + 

N-l 

+ (&» ( yn+1 ) ~ (V n+1 )) + fC yn+1 (fr n ) • (V n+1 - V n+1 ) + 

n=0 
N-l 

+ J2 7r " • i^(v n+1 ) - g,Av n+1 )) + n n ■ (g^(v n+1 ) - g^{v n+1 )). 

Now, using (|l6|), we have, for each < n < N — 1, 

tt" ■ (g^(v n+1 ) - g^{v n+1 )) + jc vn+ i(n n ) ■ (v n+1 - v n+1 ) < o 

and similarly for the terms of the second line. In the third line we have 

tt" • (g^(v n+1 ) - g,Av n+1 )) + tt" • (g^(v n+1 ) - g^(v n+1 )) < - 7 ik™ - n n \\ 2 . 

Hence 

N-l 

^2l\\n n -n n \\ 2 <0. 



n=0 

This implies tt™ = tt" for all < n < N. 

To obtain V n = V n for all < n < N, we just have to use V N — V N and apply iteratively 
the operator Q„n (V n+1 ) = V n and fo" (V n+1 ) = V n , from n = N-lton=Q. □ 

7 Convergence to equilibrium 

In this last section we discuss the main contribution of this paper, namely the exponential conver- 
gence to equilibrium for the initial-terminal value problem. Our setting is the following: consider 
a initial-terminal value problem with initial data ir~ N and terminal data V N . We will now study 
conditions under which ir° — > tt and V° — > V where (tt, V) are stationary solutions, as N — > oo. 
In fact we will show this is true if assumptions |], fj], ||, [l(] and |ll| hold. 
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7.1 A-priori bounds 

We start by establishing some useful a-priori bounds. 

Lemma 2. Suppose assumption^, and [ll] holds. Let {{■K n ,V n ) ; — N < n < N} and 

{(tt™, V n ) ; — N < n < N} be two solutions of the mean field game. Then we have 

\\V- N -V- N \\ < \\V N -V N \\+ N2K. (23) 
Proof. Applying proposition ^ we have that 

3 

Then 

g^-i(v N ) t - g^{v N )i + vf - 1 - v?- 1 <\\v N - v N i 

also we have 

j 

H6I1C6 

Exchanging the roles of (ir 1 ^ -1 , V N ) and (tt^ -1 , y^) we get 

V N-l_yN-l < ||yJV_ y JV|| +if) 

thus 

||^JV-l _ v/V-X|| < ||yiV_ v ,JV|| +i<: 

Reasoning by induction we obtain that 

|| - y-^H < \\V N -V N \\+ N2K. 

□ 

7.2 Exponential convergence 

We recover the proof of theorem ^ to obtain an important estimate: 

Proposition 13. Suppose assumptions 0, 0, [| and ^ hold. Let {(7r™, V n ) ; —A" < n < N} and 

{(tt™, V n ) ; — N < n < N} be solutions to the mean field game. Let C = I/7. Then 

N-l 

£ ||7r"-7f"|| 2 + ||V"-T/"||| 

n=-JV+l 

< C (iJtt^ - n N \\ 2 + \\V N - V^||» + \\n~ N - ^ N \\ 2 + \\V- N - V~ N \\%) 
Proof. As before, observe that 

N-l 

n=-N 
N-l 

+ ]T (7r n -7r n ).[(g # »(V rn+1 )-V n )-(g ff »(V B+1 )-V n )]. 
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Then rewriting the equation above we have 

AT-l 



(V n+1 - V n+1 ) ■ (vr™ +1 - ir n+1 ) - (V n - V n ) • (tt™ - n n ) 

71= -JV 
N—l 

= ^ tt" • (g^{v n+1 ) - g^(v n+1 )) + ic vn+ i{ir n ) ■ (v n+1 - v n+1 ) + 

n— — N 
N—l 

+ J2* n - (G*<v n+1 ) - g^{v n+1 )) +)Cy„ +1 (x n ) ■ (v n+1 - v n+1 ) + 

n=iv 
JV-l 

+ K n -(gzn(v n + 1 )-g n n(v n + 1 )) + n n -(g n n(v n+1 )-g^(v n+1 )). 



n=-JV 

Now, for each — N < n < N — 1, we have that 

7T™ ■ (g^(v n+1 ) - g„ n (y n+1 )) u y » + i(i") • (v n+1 - v n+1 ) < - 7 ||^ n+1 - v n+1 \\ 2 # , 

and similarly for the terms of the second line. In the third line we have 

tt" • (g^(v n+1 ) - g^{v n+1 )) + n n ■ (g^(v n+1 ) - g^(v n+1 )) < -tK - n n \\ 2 . 

Consequently 

JV-l 



J2 h n -n n \\ 2 + 2\\V n+1 -V n+1 \\% 

I ((^ _ -AT) . { yN _ y N) + ^-N _ ~- N) . (y-N _ y-N^ 



n=-JV 
< 



Note that, if c is the constant vector then (7r fc — 7r k ) ■ n = 0, where k = N, —N. Also, there 
exists \i k such that \\V k - V k \\ = \V k - V k + fi k \. Hence 

JV-l 

J2 h n -n n \\ 2 + \\V n -V n \\l 

n=-N+l 

N ~ JV 1 1 2 , IIT/JV T 7JVi|2 , II — JV ~ — JV 1 1 2 i I t / — JV T 7-JV||2 



< c - r + w 1 " - v n w% + \\tt-" - r + ii^~ jv - v- i — i . 

if we denote C = I/7. □ 
Define / = ||tt° - n°\\ 2 + \\V° - V°|||, and, for n > 

/„ = ||7r" - tT|| 2 + \\V n - V n \\l + \\n- n - n- n \\ 2 + \\V- n - V- n \\%. 
The previous proposition implies 

JV-l 

E fn < CfN- (24) 
n=0 

Note that the previous proposition and lemma ^| imply 



f N < h N - n N f + IK"^ - tt-^II 2 + \\V N - V N \\l + (\\V N -V N \\+ N2K) 2 . (25) 
The next lemma is the only missing tool to get exponential decay: 
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Lemma 3. Suppose f n > and that 



N-l 



J2 ^ < Wn. 



(26) 



n=0 



Then 



fo<C 



C 



N-l 



f. 



N ■ 



C + l 

Proof. The proof follows by induction. The case N = 1 is simply a particular case of (|2 

/o<C/i. 

Now, observe that 

N N i /r + iV' 1 " 1 



n=0 



/o 



n=l 
JV 



, , 1 (f) -1 

C ^1-1 



/o 



1 



C + l 

c 



1 



/o 



C + l 

c 



□ 



/io/<i. Fia; V,tt. Given N > 0, denote by 



which ends the proof. 

Theorem 7. Suppose assumptions 0, and 
(tt%, VJy) i/ie solution of the mean field game at time that has initial distribution and 
terminal cost V N = V . 
Then, as N — > oo 

where V and n is the unique stationary solution. 

Proof. By theorem [| we can define, for each N, a map Sat : S x R d /]R — > § x K d /M, that associates 
to each pair (ff , V") the pair (tt^, V# ). Here § x R d /R is given the product topology where in R /R 
we consider the norm ||C||# = inffcgR \\U + k\\. 

Now, lemma || and equation (]25|) show that, for any two pairs (V, 7r) and (V, n), we have 



as N — » oo. Thus, 
as Af — > oo. 



S JV (y,7r)-S JV (V , ,7r)-^(0,0) 
HiV (V, tt) - (F, 7f ) = E N ( V, tt) - Sjv ( V", 7f ) - (0, 0) 



□ 
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